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Abstract 


The  authors  of  this  report  appeared  on  a  panel  at  SEPG  ’99  (1999  Software  Engineering  Pro¬ 
cess  Group  Conference)  on  March  10,  1999,  in  Atlanta,  Georgia.  Each  panelist  is  one  of  the 
“most  active  lead  assessors”  in  the  Software  Engineering  Institute  (SEI)  Appraiser  Program 
and  has  conducted  and  reported  on  over  5  Capability  Maturity  Model®  (CMM®)  -Based  Ap¬ 
praisals  for  Internal  Process  Improvement  (CBA  IPIs)  over  the  past  18  months.  In  this  report, 
the  panelists  document  their  experiences  regarding  why  an  organization  chooses  to  have  a 
CBA  M  and  what  the  organization  gains  from  having  conducted  an  assessment. 

Each  author  discusses  the  experiences  related  to  CBA  IPIs  that  he  or  she  has  led  with  a  focus 
on  the  organizational  perspective  before,  during,  and  after  the  assessment.  Since  a  CBA  M  is 
a  major  organizational  intervention,  it  is  important  to  know  why  an  organization  has  an  as¬ 
sessment,  if  it  has  a  positive  impact  on  an  organization,  and  how  an  organization  would  char¬ 
acterize  the  benefits  that  the  assessment  provides.  What  needs  to  be  done  during  the  planning, 
how  the  assessment  is  conducted,  and  what  follows  the  assessment  are  all  important  factors 
that  will  affect  the  benefits  that  the  organization  experiences  from  the  assessment. 
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1  Background 


1.1  History  of  SEI  Assessments 

In  early  software  process  publications,  a  software  maturity  framework  and  questionnaire 
[Humphrey  87a]  were  developed  to  help  organizations  characterize  the  current  state  of  their 
software  practices  and  to  help  the  U.S.  Government  assess  the  ability  of  potential  Department 
of  Defense  (DoD)  contractors  to  develop  software  through  Software  Capability  Evaluations 
(SCEssm)  [Humphrey  87b].  The  Software  Engineering  Institute  (SEI)  assisted  a  number  of 
organizations  in  performing  assessments  [Olson  89]  based  largely  on  the  Maturity  Question¬ 
naire.  In  1988-91,  the  SEI  provided  training  to  organizations  that  wished  to  perform  self- 
assessments  of  their  software  processes. 

In  1990  the  SEI  commercialized  the  Software  Process  Assessment  (SPA)  to  disseminate  the 
technology  more  broadly.  Industry  and  government  licensees  were  selected  as  vendors  to 
market  assessment  services.  During  1991-1993,  SEI  self-assessment  training  was  gradually 
phased  out  and  replaced  by  vendor  training.  Based  largely  on  the  success  and  widespread  use 
of  the  maturity  framework  and  Software  Process  Assessments,  Version  1.0  of  the  Capability 
Maturity  Model®  for  Software  (CMM®)  was  published  in  1991  [Paulk  91,  Weber  91].  In  1993 
the  CMM  was  revised,  and  Version  1.1  was  published.  Various  organizations  modified  SPAs 
to  reflect  the  CMM;  however,  the  CMM-Based  Appraisal  for  Internal  Process  Improvement 
(CBA IPI)  method  is  the  first  CMM-based  assessment  method  released  by  the  SEI.  After  be¬ 
ing  field  tested  in  1994,  CBA  IPI  V  1.0  was  released  in  May  1995  and  upgraded  to  CBA  IPI 
Vl.l  in  March  1996. 

Data  from  assessments  are  collected  by  the  SEI  and  provide  information  on  the  state  of  the 
practice.  Updates  to  the  process  maturity  profile  of  the  software  community  are  published 
twice  a  year  and  are  available  on  the  SEI  Web  site: 

<URL:  http://www.sei.cmu.edu/sema/profile.html>. 

1.2  Overview  of  the  CBA  IPI  Method 

The  CBA  IPI  method  is  a  diagnostic  tool  that  enables  an  organization  to  gain  insight  into  its 
software  development  capability  by  identifying  strengths  and  weaknesses  of  its  current  proc¬ 
esses,  to  relate  these  strengths  and  weaknesses  to  the  CMM,  to  prioritize  software  improve- 
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ment  plans,  and  to  focus  on  software  improvements  that  are  most  beneficial,  given  the  or¬ 
ganization’s  current  level  of  maturity  and  its  business  goals. 

The  CBA IPI  method  is  an  assessment  of  an  organization’s  software  process  capability  by  a 
trained  group  of  professionals,  led  by  an  SEI-authorized  lead  assessor,  who  work  as  a  team  to 
generate  findings  and  ratings  relative  to  the  CMM  key  process  areas  (KPAs)  within  the  as¬ 
sessment  scope.  The  findings  are  generated  from  data  collected  from  questionnaires,  docu¬ 
ment  reviews,  presentations,  and  in-depth  interviews  with  middle  managers,  project  leaders, 
and  software  practitioners. 

The  CBA  IPI  method  has  two  primary  goals: 

1.  Support,  enable,  and  encourage  an  organization’s  commitment  to  software  process 
improvement. 

2.  Provide  an  accurate  picture  of  the  strengths  and  weaknesses  of  the  organization’s  current 
software  process,  using  the  CMM  as  a  reference  model,  and  identify  KPAs  for 
improvement. 

The  approach  of  the  CBA  IPI  method  is  to  assemble  and  train  a  competent  assessment  team 
under  the  leadership  of  a  lead  assessor  and  to  conduct  a  structured  series  of  activities  with 
key  people  in  the  organization  to  understand  their  problems,  concerns,  and  ideas  for  im¬ 
provement.  The  method  is  based  on  the  following  key  assessment  principles: 

•  Use  the  Capability  Maturity  Model  for  Software  V 1 . 1  as  a  process  reference  model. 

•  Use  a  formalized  assessment  process  that  complies  with  the  CMM  Appraisal  Framework 
(CAF)  [Masters  95]. 

•  Involve  senior  management  as  the  assessment  sponsor. 

•  Base  the  assessment  on  the  sponsor’s  business  goals  and  needs. 

•  Observe  strict  confidentiality  by  guaranteeing  that  no  information  will  be  attributed  to  an 
individual  or  project. 

•  Approach  the  assessment  as  a  collaborative  activity  between  the  assessment  team  and  the 
organizational  participants. 

The  business  needs  for  process  improvement  drive  the  requirements  for  an  assessment.  The 
business  needs  for  process  improvement  generally  include  one  or  more  of  three  closely  re¬ 
lated  factors:  reducing  costs,  improving  quality,  and  decreasing  time  to  market.  The  funda¬ 
mental  assumption  is  that  the  development  processes  largely  determine  these  factors.  More 
information  on  the  CBA  IPI  method  can  be  found  in  [Dunaway  96a], 

1 .3  Organizational  Impacts 

There  are  impacts  to  an  organization  that  take  place  as  a  result  of  conducting  an  assessment. 
This  section  is  a  composite  of  ideas  expressed  by  the  authors  of  this  report  as  well  those  lead 
assessors  included  in  the  Acknowledgments  section. 
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Specific  aspects  of  the  assessment  process  facilitate  “unfreezing”  in  an  organization  that  is 
just  beginning  a  process  improvement  effort  and  help  to  provide  momentum  to  organizations 
that  are  well  underway  with  a  process  improvement  program.  Some  key  aspects  of  the  as¬ 
sessment  process  are 

•  confidentiality  rules  that  help  build  safety  and  promote  an  accurate  understanding  of  the 
current  state  of  the  organization’s  process 

•  draft  findings  presentations  that  build  confidence  in  the  team’s  work  and  demonstrate  to 
the  organization’s  staff  members  that  they  are  being  heard 

•  clear  focus  on  the  fact  that  the  findings  are  within  the  power  of  the  organization  to 
resolve 

•  behavior  of  the  assessment  team  itself,  which  can  and  should  model  and  demonstrate 
high-level  maturity  behavior 

Placing  too  great  an  emphasis  on  the  data-gathering  and  the  data-management  aspects  of  as¬ 
sessments,  to  the  neglect  of  these  key  interventional  aspects,  can  reduce  the  chances  of  ap¬ 
propriate  process  improvement  follow-on  action.  On  the  other  hand,  a  proper  balance  be¬ 
tween  data  handling  and  organizational  intervention  increases  the  chances  of  appropriate 
improvement  actions. 

The  nature  of  an  organization’s  business  is  an  important  consideration.  Is  the  organization  a 
research  and  development  center,  a  DoD  contractor,  a  large  software  product  developer,  or  a 
consulting  organization?  Has  the  assessment  been  mandated  by  a  parent  organization,  or  is  it 
the  result  of  an  internal  initiative? 

Other  important  factors  that  will  affect  the  assessment  include 

•  the  size  of  the  organization  being  assessed 

•  whether  there  are  any  previous  or  ongoing  quality  initiatives 

•  the  amount  of  CMM  training  and  experience 

•  how  the  method  is  tailored  to  meet  the  organization’s  business  goals  (The  assessment 
method  provides  many  opportunities  for  tailoring.  How  the  method  is  tailored  will  have 
an  important  impact  on  the  process  culture.) 

1 .4  User  Feedback 

Presentations  were  given  at  SEPG  ’96  [Dunaway  96b]  and  SEPG  ’98  [Dunaway  98]  to  report 
the  feedback  that  was  provided  to  the  SEI  regarding  the  use  of  the  CBA 1PI  method. 

Panelists  from  previous  conferences  who  shared  their  experiences  and  results  achieved  by  use 
of  the  CBA  IPI  method  are  listed  in  the  Acknowledgments  section. 
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2  Perspectives  on  Assessments 


The  following  individuals  appeared  on  a  panel  at  SEPG  ’99  on  March  10,  1999,  in  Atlanta, 
Georgia,  to  share  their  experiences  with  the  audience  in  the  use  of  the  CBA IPI  assessment 
method.  They  were  asked  to  record  their  comments  in  this  technical  report  so  that  people  who 
were  unable  to  attend  the  conference  could  benefit  from  their  experiences. 

2.1  Panelist:  Ruth  Berggren,  Electronic  Data  Systems 

2.1.1  Why  Do  EDS  Organizations  Conduct  Assessments? 

To  answer  this  question,  it  may  be  helpful  first  to  look  briefly  at  the  historical  background  of 
process  improvement  at  Electronic  Data  Systems  (EDS). 

In  1987  the  corporation  rolled  out  a  standard  methodology  called  Systems  Life  Cycle  (SLC) 
and  established  an  expectation  that  all  organizations  would  use  it  to  deliver  products  and 
services  to  customers.  The  methodology  was  supported  by  education,  detailed  documenta¬ 
tion,  and  later  an  assessment  and  implementation  process.  By  1993  awareness  of  the  CMM 
had  grown  significantly,  and  EDS  adopted  the  CMM  for  Software  V  1.1  as  a  standard  for 
process  improvement  and  performance  for  software  development  and  maintenance.  By  1996 
the  first  EDS  organization  achieved  the  first  Level  2  rating  (we’ve  had  about  35  more  since 
then),  and  by  1998  we  achieved  the  first  Level  3  rating.  Most  of  the  CMM-based  process  im¬ 
provement  initiatives  in  EDS  were  predicated  by  SLC  implementations,  which  helped  move 
the  corporate  culture  toward  the  use  of  processes  to  improve  effectiveness  and  efficiency. 
EDS  currently  is  experiencing  a  large  increase  in  the  number  of  assessments  each  year,  as 
more  organizations  move  forward  with  process  improvement. 

The  increase  in  assessments  and  process  improvement  initiatives  is  driven  primarily  by  the 
customer  in  the  form  of 

•  pressure  to  achieve  higher  levels  of  maturity  to  keep  the  business  that  we  have  with  them 
or  to  acquire  new  or  additional  business 

•  extra  bidding  points  awarded  for  higher  levels  of  maturity  for  DoD  contracts 

•  requirements  for  higher  levels  of  maturity  to  bid  as  prime  contractor  for  DoD  contracts 

The  bottom  line  is  that  most  assessments  in  EDS  are  performed  to  provide  authentication  of 
our  claims  of  process  capability  and  process  maturity.  Many  EDS  organizations  initially 
“pushed”  by  customer  pressure  to  achieve  Level  2  begin  to  see  the  benefits  and  continue  the 
journey. 
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What  benefits  are  realized? 


•  Project  performance  is  improved,  in  terms  of  fewer  defects  delivered  in  the  software;  this 
results  in  more  time  to  spend  on  providing  creative  solutions  for  the  customer  and 
reduced  cost.  Our  Level  2  projects  also  deliver  on  time  more  consistently. 

•  Employee  morale  improved  in  most  cases  because  of  the  involvement  of  the  organization 
in  the  assessments.  The  feedback  of  assessment  findings  and  subsequent  action  on  those 
findings  leads  to  more  focused  improvements. 

•  Most  customers  appreciate  the  improvements  in  communication  and  project  results. 

•  Organizations  that  had  successfully  implemented  the  corporate  SLC  methodology  were 
able  to  implement  the  CMM  more  easily  and  were  generally  easier  to  assess.  ■ 

•  Assessment  team  members  learn  the  CMM,  gain  process  improvement  insight,  and  carry 
that  knowledge  and  insight  to  the  process  improvement  team.  These  team  members  are 
used  in  other  assessments  in  other  corporate  divisions.  The  organization  also  learns  a 
great  deal  about  the  CMM  and  about  itself. 

This  is  not  to  say  that  we  do  not  experience  some  difficulties  as  well.  We  occasionally  have  a 
sponsor  whose  objectives  for  doing  the  assessment  are  not  closely  aligned  with  the  objectives 
documented  in  the  method.  This  condition  results  in  institutionalization  difficulties  as  well  as 
difficulties  in  conducting  the  assessment  due  to  inadequate  site  preparation  or  rushing  to  get 
the  assessment  done  before  the  organization  is  ready.  Other  organizations  aim  too  low;  they 
have  no  strategic  decision  to  attain  Level  5.  In  most  organizations,  measurement  is  the  most 
significant  institutionalization  problem. 

2.1 .2  Do  Assessments  Pay  Off  for  EDS? 

The  answer  is  a  resounding  Yes\  Even  when  we  have  difficulties,  we  learn  a  great  deal  from 
each  assessment.  These  are  just  a  few  of  the  things  we  have  learned: 

•  Sponsorship  is  the  most  critical  factor  for  the  success  of  an  assessment  and  process 
improvement  effort.  Get  the  right  sponsor,  and  cultivate  others. 

•  Organization  goals  and  objectives  must  be  tied  to  a  strategic  business  plan — without  it 
you  can  end  up  in  insanity.  The  business  plan  usually  identifies  where  the  money  will  be 
spent.  The  average  direct  cost  of  a  CBA IPI  assessment  is  $45,000-50,000. 

•  All  that  you  hear  about  change  management  applies  to  CMM  implementation  (education, 
vision,  measurement,  etc.).  Put  your  best  people  on  implementation  and  assessment 
teams.  Send  them  to  SEI  classes,  the  SEI  Symposium,  and  the  SEPG  Conference. 

•  Everyone  leams  (even  the  ones  that  are  dragged  in  kicking  and  screaming).  They  learn 
about  the  organization,  the  CMM,  and  about  themselves.  In  most  cases  it’s  a  positive 
experience  for  everyone. 

•  Assessments  help  move  the  organizational  culture  toward  measuring  improvement. 

•  Assessments  galvanize  the  organization  to  move  forward  with  process  improvement. 


6 


CMU/SEI-99-TR-01 2 


2.2  Panelist:  Gilles  des  Rochettes,  Thomson-CSF 

2.2.1  The  Thomson-CSF  Context 

Thomson-CSF  is  in  the  business  of  professional  electronic  equipment  and  systems.  It  is  com¬ 
posed  of  the  following  eight  independent  business  groups,  each  of  which  is  composed  of 
business  units: 

1 .  Aerospace 

2.  Avionics 

3.  Communication 

4.  Naval  Combat  Systems 

5.  Information  Systems  and  Services 

6.  Optronics 

7.  Air  Traffic  Management  and  Missiles  Systems 

8.  Tubes  and  Components 

The  company,  when  responding  to  several  requests  for  proposal,  had  to  deal  with  the  re¬ 
quirements  in  the  CMM  for  Software  V  1.1  (Level  2  or  Level  3)  through  SCEs  or  audits  by 
the  Federal  Aviation  Administration  (FAA),  Ministries  of  Defence  (MoD)  (Canada,  UK, 
Australia,  etc.),  Civil  Administration,  prime  contractors,  etc. 

While  each  business  group  and  business  unit  is  independent  for  creating  business,  there  are 
many  networks  called  common  efficiency  teams  (CETs)  dedicated  to  the  key  technical,  in¬ 
dustrial,  and  purchase  areas  (such  as  software,  systems  engineering,  antennas,  signal  proc¬ 
essing,  and  microelectronics).  Their  missions  are  to  share  experiences,  assets,  common  de¬ 
velopments,  studies,  resources,  and  negotiations  with  suppliers.  There  is  a  Software  CET,  in 
which  the  SPICE-Th  corporate  action  (not  related  to  ISO- 15504  but  to  the  SEI’s  CMM  for 
Software)  is  acting  with  its  sub-network,  the  Thomson-CSF  software  process  improvement 
network  (SPIN).  Within  the  Software  CET,  the  SPICE-Th  corporate  action  organizes  all  of 
the  official  CBA IPI  assessments  and  mini-assessments  to  support  the  business  groups’  and 
business  units’  software  engineering  process  group  (SEPG)  staffs.  For  assessments,  Thom¬ 
son-CSF  works  with  two  U.S.  lead  assessors  and  three  internal  lead  assessors  (two  others  will 
soon  be  authorized).  There  is  also  a  pool  of  more  than  50  CBA  IPI  assessment  team  mem¬ 
bers.  This  allows  Thomson  to  have  not  only  an  assessment  program  where  assessments  are 
done,  but  also  a  corporate  program  that  upgrades  the  team  members’  knowledge  and  experi¬ 
ence  in  the  CMM,  particularly  for  high-level  assessments.  Incentives  are  awarded  each  year 
during  a  one-day  celebration,  which  is  another  opportunity  to  share  experience  and  new  in¬ 
formation.  There  is  an  annual  rate  of  9  official  CBA  IPI  assessments  and  15  mini-assessments 
per  year. 

A  Systems  Engineering  CET  has  a  pool  of  15  assessment  team  members.  Usually,  SE-CMM 
assessments  and  hardware  assessments  are  conducted  using  the  CBA  IPI  method  led  by  soft- 
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ware  lead  assessors,  using  some  transpositions  of  the  CMM  through  domain-specific  scripts. 
A  policy  was  issued  in  1994  by  the  chief  executive  officer  (CEO)  of  Thomson-CSF,  requiring 
that  all  business  units  be  at  Level  3  for  software;  this  provides  a  strong  impetus  for  all  busi¬ 
ness  units.  On  the  systems  engineering  side,  most  of  the  formal  CBA IPI  assessments  are 
followed  with  the  use  of  a  questionnaire-based  assessment,  a  mini-assessment,  or  a  delta  as¬ 
sessment.  Thomson-CSF  is  looking  forward  to  using  the  Integrated  CMM  for  Software  and 
Systems  Engineering  (CMMI-SE/SW)  and  having  a  model  for  systems  engineering  with  a 
recognized  status.  We  will  gain  leverage  from  integrated  assessments  with  the  software  staff 
who  will  bring  their  experience  and  momentum. 

2.2.2  Credibility  of  the  Assessment  Team 

The  key  ingredient  of  a  successful  assessment  is  to  get  consensus  from  the  interviewees  on 
the  areas  of  improvement.  Consensus  is  strongly  linked  with  two  elements:  (1)  the  assess¬ 
ment  process  where  interviewees  feel  that  they  are  being  heard,  and  (2)  the  credibility  of  the 
assessment  team  so  that  interviewees  trust  the  accuracy  of  the  findings  and  agree  with  how 
the  findings  are  stated. 

What  are  the  risks  for  the  assessment  team’s  credibility?  Assessment  is  a  process,  so  it  has  to 
be  fully  controlled,  even  when  unforeseen  events  occur.  Thanks  to  our  very  close  collabora¬ 
tion  with  U.S.  lead  assessors  (we  started  in  1992),  we  have  defined  a  standard  process  for 
assessments,  with  quantitative  data,  tailoring  guides,  and  coverage  indicators.  First,  it  is  nec¬ 
essary  to  do  quantitative  process  control  of  each  of  the  tasks  during  the  on-site  period.  We 
now  have  enough  data  to  permit  that  control.  Maturity  model  knowledge  and  experience  by 
the  team  members  is  another  key  issue,  and  there  must  be  maximum  use  of  professional 
judgment.  The  way  that  questions  are  raised  and  the  content  of  the  questions  are  also 
important  factors.  The  wording  of  the  strengths  and  weaknesses  is  important,  as  well:  they 
must  not  be  too  general,  and  they  must  be  precisely  worded,  using  site  terms  and  acronyms, 
and  quoting  as  many  examples  as  possible  (e.g.,  site  standards,  assets,  events).  In  addition,  it 
is  important  to  consider  coverage  of  the  CMM,  as  well  as  the  organization  and  the  life  cycle. 
How  much  credibility  would  an  assessment  team  have  if  they  missed  some  notable  weak¬ 
nesses  for  a  project?  At  Thomson-CSF,  we  track  the  coverage  through  daily  quantitative  in¬ 
dicators  that  allow  us  to  optimize  and  replan  the  assessment  schedule.  We  also  noticed  that 
the  recommendation  report  was  a  risk  for  the  assessment  team’s  credibility.  This  can  be  a 
danger  if  you  only  turn  the  weaknesses  into  the  positive  form  or  provide  suggestions  that  do 
not  help.  We  have  chosen  to  include  only  the  top-10  recommendations  in  our  report,  but  re¬ 
quire  that  the  organization  address  all  of  the  other  weaknesses  that  were  discovered.  Obvi¬ 
ously  it  is  the  lead  assessor’s  responsibility  to  maintain  the  assessment  team’s  credibility 
during  the  whole  assessment  process  including  the  follow-on  actions. 

2.2.3  Key  Planning  Issues 

The  efficiency  of  the  interviews  is  strongly  linked  to  the  confidence  of  the  participants.  There 
are  always  barriers  to  open  discussion  for  many  interviewees,  dependent  upon  their  level  of 
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security  in  the  organization,  their  trust  in  the  assessment  team,  their  fears  for  the  future  (e.g., 
re-organizations,  actions  for  improvement).  For  example,  it  is  always  interesting  to  analyze 
why  it  is  sometimes  difficult  or  not  natural  to  identify  good  practices.  Do  the  interviewees 
fear  that  there  will  be  few  acknowledgments  or  incentives  for  what  they  create?  Do  they  fear 
that  now  their  private  practices  will  become  compulsory?  Do  they  underestimate  their  prac¬ 
tices?  The  choice  of  representatives  for  the  assessed  organization  is  key.  It  is  always  a  com¬ 
promise  between  getting  maximum  information  in  a  limited  time  and  the  risk  of  “silences”  in 
the  interviews.  There  is  also  a  risk  of  an  SEPG  leader  or  a  member  of  a  software  quality  as¬ 
surance  (SQA)  group  who  has  a  vested  interest  in  the  assessment  outcomes.  In  our  assess¬ 
ments  (apart  from  those  dealing  with  “product  line  approach”),  there  is  usually  only  one  rep¬ 
resentative  from  a  particular  group. 

2.2.4  Follow-On  Actions 

Our  first  experience,  in  1992  and  1993  when  we  did  SPAs,  produced  a  recommendation  re¬ 
port  more  than  four  months  after  the  assessment  that  was  so  thick  and  confidential  that  very 
few  people  could  read  and  use  it.  When  the  CBA IPI  method  was  released  in  early  1995,  we 
decided  to  provide  a  recommendation  presentation  no  later  than  one  month  after  the  assess¬ 
ment.  Presently  most  of  the  assessed  units  volunteer  for  a  review  of  their  strategic  plan  by 
corporate  management  and  by  the  most  advanced  units.  Currently  two  mini-assessments  are 
planned:  the  first,  which  is  planned  about  six  months  after  the  strategic  plan  is  approved,  will 
provide  an  initial  checkpoint  about  its  implementation  and  about  the  key  choices;  the  second, 
which  is  planned  about  six  months  before  the  formal  assessment,  will  be  performed  to  con¬ 
firm  the  readiness  and  the  date  of  the  assessment.  Generally  our  mini-assessments  are  done 
by  two  people  for  two  days.  We  can  assess  approximately  six  to  eight  key  process  areas  and 
read  many  documents,  plus  observe  some  demonstration  of  the  infrastructure.  There  is  also 
time  allowed  for  a  draft  finding  presentation.  For  a  small  organization  (generally  less  than  20 
people),  which  cannot  afford  the  full  assessment  method,  the  mini-assessments  provide  a 
mechanism  to  assess  the  Level  2  key  practices  and  provide  a  light  improvement  plan. 

2.2.5  On-Site  Difficulties 

We  can  share  two  difficulties  that  we  regularly  experience  during  our  assessments. 

First,  why  the  assessment  is  requested  can  be  a  problem.  Does  management  really  intend  to 
improve?  When  you  do  an  assessment,  expectations  in  the  organization  are  raised,  so  you  are 
at  risk  if  there  are  no  follow-on  actions. 

In  addition,  the  comeback  will  be  very  difficult.  We  have  units  whose  objectives  are  “the 
gold  medal.”  It  is  very  difficult  to  change  these  objectives:  at  the  corporate  level,  our  mission 
is  to  say,  “You  are  not  yet  ready,”  and  there  has  been  very  little  resistance.  Our  mission  is 
also  to  improve  their  way  of  approaching  and  interpreting  the  CMM,  and  here  we  are  not  al¬ 
ways  successful. 
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Second,  even  if  it  is  not  jeopardizing  the  assessment,  it  is  sometimes  difficult  to  get  real  con¬ 
fidentiality  from  the  interviewees  when  assessing  a  level.  Generally  lack  of  confidentiality 
occurs  between  interviewees  and  their  managers,  usually  after  the  draft  findings  presenta¬ 
tions.  We  have  noticed  that  this  is  often  the  case  with  the  SQA  staff,  who  seem  to  play  the 
role  of  “protectors  of  the  organization,”  as  they  do  when  there  are  audits  from  the  customers 
or  from  the  Defense  Administration.  We  have  not  yet  been  able  to  determine  whether  there  is 
a  real  negative  impact  on  the  quality  of  interviews  and  on  the  buy-in  by  the  interviewees. 

2.2.6  Responding  to  the  Organization’s  Business  Goals 

The  real  opportunity  to  respond  to  the  organization’s  business  goals  is  when  you  propose  the 
recommendations  and  complete  the  final  findings  report.  We  try  to  determine  the  concerns  of 
the  interviewees  and  the  management:  What  are  the  priorities  regarding  the  assessed  KPA, 
even  those  of  the  lower  levels?  What  are  the  main  problems  to  solve  (e.g.,  low  performance 
of  processes,  real-life  problems  to  deal  with,  specific  customers  to  work  with)?  Setting  pri¬ 
orities  in  this  way  will  allow  the  assessment  team  to  provide  a  top-10  list  of  recommenda¬ 
tions  that  addresses  these  concerns  both  for  determining  which  KPA  to  improve  and  for  de¬ 
scribing  how  to  achieve  a  higher  level  of  maturity. 

For  most  of  our  advanced  units,  a  return  on  investment  higher  than  four  has  been  achieved, 
even  while  getting  to  Level  2.  The  next  step  for  Thomson-CSF  is  to  do  integrated  assess¬ 
ments  and  improvements  of  the  three  key  disciplines:  contract  program  management,  system 
engineering,  and  software  engineering.  We  have  noticed  that  there  is  a  risk  of  a  maturity  gap 
between  software  and  the  other  disciplines.  That  is  why  Thomson-CSF  is  involved  in  the  ca¬ 
pability  maturity  model  integration  (CMMI)  effort  and  is  a  candidate  for  being  a  pilot  using 
the  CMMI-SE/SW. 

2.3  Panelist:  Paul  Iredale,  Reuters 

2.3.1  Software  Process  Improvement  History  at  Reuters 

At  the  beginning  of  1996,  Reuters  piloted  its  first  process  improvement  initiatives  in  various 
development  groups  worldwide  based  on  an  approach  used  by  Hewlett  Packard.  A  develop¬ 
ment  group  was  asked  sets  of  questions  to  get  an  understanding  of  the  major  problems  facing 
the  organization  at  that  time.  The  results  of  these  questions  were  presented  to  the  organization 
as  a  set  of  opportunities  not  unlike  the  CMM  formal  assessment  currently  used  (CBA IPI).  A 
process  improvement  project  was  then  set  up  to  address  the  most  critical  problem.  It  soon 
became  apparent  that,  by  the  time  the  project  delivered  the  process,  there  were  other  new, 
more  pressing  problems  and  that  the  expected  benefits  of  the  specific  process  improvement 
project  had  significantly  diminished.  So  in  August  1996,  a  decision  was  made  to  employ  a 
standard  to  guide  organizations  in  the  Development  Federation  in  their  process  improvement 
activities.  The  SEI  Capability  Maturity  Model  was  chosen  as  the  standard.  Goals  were  set  for 
groups  to  achieve  CMM  Level  2  by  the  end  of  1998  and  CMM  Level  3  by  the  end  of  1999. 
On  December  4,  1998,  the  final  group  in  the  Corporate  Technology  Group  (CTG)  software 
process  improvement  initiative  achieved  CMM  Level  2.  The  UK  Operations  Group,  which 
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consists  of  68  people  in  3  support  groups,  was  told  of  their  achievement  1  year,  1  month,  and 
17  days  after  the  first  CTG  Group  [Equisoft  in  Colchester  (now  CSTG-UK)]  achieved  their 
Level  2  rating.  The  Transactions  Product  Groups  in  Hauppauge  achieved  Level  2  in  Septem¬ 
ber  1997,  and  the  Transactions  Product  Group  in  London  achieved  Level  2  in  October  1998. 

In  total  over  1500  software  engineers  in  14  locations  and  5  countries  all  are  at  CMM  Level  2. 
Now  the  goal  is  to  achieve  Level  3  in  1999. 

2.3.2  Why  Have  Assessments? 

The  CMM  offers  a  framework  for  continuous  process  improvement  that  is  deployed  in  a  se¬ 
ries  of  steps,  or  maturity  levels.  Each  maturity  level  builds  upon  the  previous  level  and  the 
organization  achieves  known  operational  characteristics  as  maturity  increases.  The  attraction 
of  this  approach  is  the  common  framework  and  language,  coupled  with  the  local  interpreta¬ 
tion  needed  to  align  the  process  with  the  differing  business  needs  of  the  development  groups. 
In  simple  terms,  you  can  think  globally  and  act  locally.  The  CMM  as  a  process  improvement 
framework  directs  the  organization  by  indicating  what  needs  to  be  done  to  achieve  the  ma¬ 
turity  levels,  and  the  local  organization  makes  the  decision  regarding  how  activities  are  im¬ 
plemented.  This  approach  allows  the  common  framework  to  be  molded  to  the  local  needs  of 
the  organization’s  business. 

Assessments  raise  awareness  within  groups  by  mirroring  the  organization,  its  strengths,  and 
its  weaknesses.  Groups  take  ownership  of  their  problems  and  plan  to  resolve  them  in  a  logical 
order  that  is  indicated  by  the  model.  Internally  proposed  improvements  are  accepted,  rather 
than  resisted.  An  assessment  is  a  logical  part  of  the  IDEALSM  improvement  cycle;  it  provides 
the  ability  to  look  jointly  into  the  mirror  and  openly  agree  on  the  problems.  It  also  helps 
maintain  a  continuous  focus  on  software  process  improvement  (SPI),  even  in  places  where 
sponsorship  is  light. 

2.3.3  Assessment  Mix 

Performing  formal  assessments  such  as  CBA  IPIs  every  12  to  24  months  is  not  sufficient.  We 
also  use  mini-assessments  every  quarter  and  a  scoring  system  that  is  reported  monthly  to 
track  SPI  progress  very  closely.  This  is  analogous  to  major  project  milestones  (quarterly 
mini-assessments)  and  regular  monitoring  of  tasks  in  progress  (monthly  scores).  Major  mile¬ 
stones  always  provide  evidence  of  solid  improvement  through  greater  depth  or  review. 


SM  IDEAL  is  a  service  mark  of  Carnegie  Mellon  University. 
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2.3.4  What  Benefits  Has  Achieving  CMM  Level  2  Brought  to 
Reuters  ? 

The  most  obvious  benefits  that  have  come  from  achieving  Level  2  are  concerned  with  man¬ 
agement  and  control  of  the  development  projects  being  run  within  the  group.  Management 
now  has  a  clear  view  of  the  cost,  effort,  and  resources  required  to  perform  projects  within 
their  groups.  In  addition,  the  development  organizations  have  started  to  measure  key  pa¬ 
rameters  related  to  their  performance,  and  these  measurements  are  being  used  to  identify  ar¬ 
eas  where  the  process  can  be  made  more  efficient.  These  benefits  are  the  tangible  ones  that 
directly  relate  to  a  process  being  deployed  within  the  groups. 

There  are  other  benefits  that  are  more  subtle,  but  of  greater  importance.  Adopting  the  CMM 
provides  a  mechanism  for  introducing  global  change  into  the  Reuters  development  commu¬ 
nity.  As  higher  maturity  levels  are  reached,  corporate  goals  are  used  to  steer  the  process  im¬ 
provement  activities;  this  aligns  the  process  and  technology  with  the  direction  of  the  corpo¬ 
rate  business  strategy. 

2.3.5  How  Have  the  Development  Groups  Changed  as  a 
Result  of  Their  Process  Improvement  Activities? 

There  is  now  a  structure  of  software  engineering  process  groups  and  councils  (SEPGs  and 
SEPCs).  The  SEPG  would  typically  contain  staff  representing  the  main  disciplines  within  the 
group:  architecture,  design,  coding,  test,  software  quality,  project  management,  and  training. 
Each  SEPG  has  one  representative  sitting  on  the  area  SEPC.  This  network  of  process  im¬ 
provement  groups  has  already  started  to  look  at  some  of  the  common  issues  of  CMM  Level  3 
by  setting  up  further  groups  to  address  issues  of  importance  in  an  area,  and  sometimes  on  a 
global  basis.  Some  examples  of  these  groups  and  their  activities  are 

•  the  Global  Testing  Association  (GTA),  which  seeks  to  provide  a  common  testing 
approach  for  all  groups 

•  the  UK  SQA  working  group 

•  the  North  American  Training  Council  (NATC) 

•  the  United  Kingdom  Training  Council  (UKTC) 

The  training  councils  link  with  human  resources  (HR)  initiatives  in  staff  development  to  pro¬ 
vide  a  curriculum,  a  career  map,  and  a  training  plan  for  addressing  the  skills  required  for 
Level  3  activities.  Development  groups  have  started  to  take  responsibility  for  specific  areas 
of  the  Level  3  initiative,  and  these  groups  will  be  reviewing  their  ideas  with  other  groups 
worldwide,  with  the  intention  of  providing  a  solution  that  everyone  can  use.  This  focus  is 
now  causing  groups  to  communicate  with  each  other  on  a  regular  basis  on  technical  and  pro¬ 
cess  issues  using  a  common  language  to  describe  the  process  and  technology.  The  individuals 
involved  in  these  initiatives  are  motivated  to  improve  their  working  environment.  The  CMM 
initiative  has  empowered  them,  and  they  are  beginning  to  make  changes  that  will  have  a  pro¬ 
found  effect  on  the  way  that  development  is  performed.  The  CMM  is  providing  an  infra- 
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structure  for  groups  to  communicate  with  and  understand  each  other,  and  is  giving  individu¬ 
als  the  opportunity  to  improve  their  working  practices. 

2.4  Panelist:  Itzhak  Lavi,  Israel  Aircraft  Industries 

Israel  Aircraft  Industries  (IAI)  is  the  largest  industrial  enterprise  in  Israel.  It  is  composed  of 
14  divisions,  of  which  6  are  heavily  involved  in  development,  production,  and  maintenance 
of  embedded  computer  systems. 

IAI  started  to  promote  corporate-level  activities  for  improving  its  software  development 
methods  in  the  early  80s,  and  later  (a  short  time  after  the  publication  of  the  CMM)  adopted 
the  CMM  as  a  reference  for  improving  the  software  processes  employed  in  IAI’s  divisions. 

IAI  has  performed  12  software  assessments  from  1992  through  the  end  of  1998.  Three  of 
these  assessments  were  “initial  assessments,”  while  the  other  nine  were  “non-initial  assess¬ 
ments.”  Initial  assessments  are  conducted  as  a  first  step  in  an  organization  that  is  just  begin¬ 
ning  to  improve  its  software  process.  Non-initial  assessments,  on  the  other  hand,  are  con¬ 
ducted  mainly  to  confirm  that  a  pre-defined  milestone  of  achieving  of  a  certain  CMM  level 
has  been  attained.  It  is  possible  for  non-initial  assessments  to  be  the  first  assessment  in  an 
organization,  since  an  organization  may  start  a  SPI  program  without  a  formal  initial  assess¬ 
ment. 

Israel  Aircraft  Industries  started  to  perform  assessments  in  1992,  a  short  time  after  its  SPI 
program  was  launched.  The  first  assessments  were  initially  intended  to  identify  major  defi¬ 
ciencies  (and  strengths)  in  IAPs  software  processes  and  to  set  priorities  for  corrective  actions. 
However,  the  main  purpose  was  to  get  senior  management’s  attention  and  sponsorship,  as 
well  as  the  wide  participation  of  middle  managers  and  practitioners  in  the  SPI  efforts. 

The  first  two  assessments  conducted  in  1992  were  Software  Process  Assessments,  and  they 
took  place  in  two  different  divisions.  The  results  were  only  partially  successful;  although  they 
both  gave  reasonably  good  results  in  identifying  and  prioritizing  areas  for  improvement,  only 
one  of  these  assessments  was  successful  in  the  getting  management’s  attention  and  wide  buy- 
in.  In  fact,  in  one  of  the  divisions  where  senior  management  was  skeptical  from  the  begin¬ 
ning,  the  assessment  results  did  not  result  in  a  major  change. 

Later,  when  our  processes  began  to  improve,  we  conducted  more  assessments,  most  of  them 
CBA  IPIs,  mainly  to  verify  that  the  desired  CMM  level  was  achieved,  but  also  to  identify 
further  deficiencies  and  to  set  the  framework  for  further  efforts  in  process  improvement. 

We  found  that  the  high  price  of  formal  CBA  IPI  assessments  (both  in  cost  and  in  disturbance 
to  the  assessed  organization)  does  not  allow  for  such  assessments  to  be  performed  more  than 
once  every  two  to  three  years  in  the  same  division.  We  also  found  that  assessments  performed 
to  verify  attainment  of  a  CMM  level  may  not  be  the  best  tool  for  detecting  deficiencies  in  the 
software  process.  For  this  purpose,  we  use  informal  process  audits.  While  the  CBA  IPI 
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method  is  well  defined  and  is  based  on  firm  principles  that  guarantee  the  accuracy  of  the  as¬ 
sessments  results,  our  informal  process  audits  are  based  mainly  on  our  own  experience.  We 
would  like  to  have  a  more  instructive  and  well-defined  method  for  performing  these  audits — 
a  method  that  would  be  based  on  wide  experience  and  meticulous  analysis  of  the  information 
that  is  obtained  in  the  audit  and  its  interpretation. 

In  general,  assessments  do  pay  off  when  they  are  performed  as  an  integral  part  of  a  SPI  effort. 
An  assessment  costs  around  $100,000,  including  the  preparation  and  team  training,  but  this  is 
only  a  small  fraction  of  the  cost  of  an  ongoing  SPI  effort.  There  is  not  much  data  on  the  re¬ 
turn  on  investment  (ROI)  of  SPI  efforts.  I  believe  the  main  reason  for  this  is  that  it  is  very 
difficult  to  measure  the  quantitative  outcome  of  such  efforts,  due  to  the  following  reasons: 

•  Organizations  with  a  low  maturity  lack  a  reliable  database  of  measurements. 

•  Process  improvement  is  a  lengthy  process.  With  the  quickly  changing  software 
development  technology,  a  few  years  may  also  mean  changes  in  the  software  staff, 
changes  in  the  development  environment  (hardware,  software  and  network),  changes  in 
the  methods  used  (e.g.,  transfer  to  object-oriented  design,  or  OOD),  and  changes  in  the 
programming  languages  used.  It  is  not  clear  to  what  extent  measured  results  are  the 
outcome  of  process  improvement  efforts,  rather  than  the  other  changes. 

However,  at  IAI,  we  have  observed  some  significant  improvement  in  our  performance,  such 
as  a  significant  decrease  in  system-integration  time  and  in  the  effort  required  to  correct  soft¬ 
ware  defects. 

In  addition,  the  following  non-quantitative  results  of  a  SPI  program  have  been  observed: 

•  awareness  of  software  management  practices 

•  better  understanding  of  the  engineering  nature  of  software  development 

•  institutionalization  of  software  engineering  process  groups  in  IAI’s  divisions 

•  construction  of  a  general  de  facto  model  for  process  improvement  in  the  enterprise 

•  increase  in  marketing  capabilities 

To  summarize,  based  on  IAI’s  accumulated  experience,  the  following  observations  and  con¬ 
clusions  can  be  made: 

•  Initial  assessments  pay  off,  provided  that  senior  management  is  willing  to  listen  and  use 
their  results  seriously.  In  this  case,  they  can  help  in  unfreezing  organizations.  Although 
CBA IPI  assessments  provide  more  accurate  results,  SPAs  can  be  used  as  initial 
assessments,  since  they  are  less  expensive. 

•  Initial  assessments  are  not  always  required.  This  may  be  the  case  when  the  present 
process  deficiencies  and  strengths  are  already  known,  senior  management  is  already, 
convinced  that  the  software  process  should  be  improved,  and  they  are  ready  to  direct  the 
organization  to  make  this  effort. 

•  In  other  cases,  senior  management  may  not  really  be  interested  in  improving  the  software 
process.  In  such  cases,  initial  assessments  may  not  be  sufficient  to  start  a  SPI  effort. 
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•  Non-initial  assessments  also  pay  off  when  they  are  performed  to  verify  that  a  major  SPI 
program  milestone  (such  as  “achieve  level  X  by  a  certain  date”)  has  been  attained.  A 
CBA IPI  assessment  usually  will  produce  results  that  reflect  quite  accurately  the  status  of 
an  organization’s  software  process.  However,  they  should  not  be  done  more  frequently 
than  once  every  two  to  three  years  in  the  same  organization  since  they  are  rather 
expensive. 

•  Using  assessments  both  for  level  rating  (when  a  certain  level  is  expected)  and  as  a 
primary  tool  for  identifying  deficiencies  is  risky,  as  it  may  lead  to  less  reliable  results. 
Efforts  should  be  made  to  develop  a  reliable  method  for  conducting  process  audits 
(“mini-assessments”)  that  will  identify  deficiencies  and  strengths  without  attempting  to 
determine  the  CMM  level.  Such  “mini-assessments”  should  be  much  less  expensive  and 
should  be  conducted  two  to  four  times  a  year  to  provide  an  ongoing  mechanism  for 
setting  the  proper  framework  for  the  SPI  efforts. 

•  While  it  is  difficult  to  calculate  the  quantitative  results  (ROI)  of  a  SPI  program,  there  are 
many  significant  non-quantitative  results  that  can  be  associated  with  such  an  effort. 

2.5  Panelist:  Guy  Taylor,  U.S.  Navy 

At  the  Naval  Surface  Warfare  Center,  Port  Hueneme  Division,  Dam  Neck,  our  involvement  in 
software  process  improvement  began  as  an  SEI  affiliate  in  1990.  Our  organizations  served  as 
beta  test  sites  for  the  Maturity  Questionnaire  in  1992  and  have  hosted  the  SEI  on  several 
other  occasions.  Our  first  exposure  to  the  CMM  and  assessments  occurred  in  1992  when  a 
Software  Capability  Evaluation  was  performed  on  our  organization  as  we  were  competing  for 
a  joint  tactical  software  system  development  project.  The  evaluation  team  informed  us  after 
the  assessment  that  we  had  everything  needed  to  be  a  Level  2  organization  except  for  a 
quality  assurance  plan. 

At  that  point,  our  organization  began  the  journey  in  process  improvement.  Our  organization 
obtained  CMM  training  and  formed  a  team  to  conduct  SCEs.  A  part-time  software  engineer¬ 
ing  process  group  was  established,  and  the  team  began  performing  internal  assessments  in 
1994.  (Note:  We  consider  an  internal  assessment  as  one  that  is  performed  by  a  team  where  all 
members  are  employees  of  our  local  detachment,  including  the  lead  assessors.  An  external 
assessment  is  one  that  is  performed  by  a  team  that  is  led  by  and  includes  members  from  a 
commercial  contractor.) 

In  1995,  a  team  of  six  people  received  lead  assessor  training  to  facilitate  our  process  im¬ 
provement  efforts.  We  established  a  full-time  SEPG,  built  our  process  asset  library,  and  im¬ 
plemented  a  key  process  owner  concept.  The  lead  assessors  then  performed  internal  assess¬ 
ments  using  the  CBA  IPI  method.  We  used  that  method  to  verify  our  Level  2  rating,  and  we 
continued  performing  internal  assessments  focusing  on  new  projects  to  determine  their  Level 
2  conformance.  Following  a  major  reorganization,  another  internal  assessment  was  per¬ 
formed  to  verify  that  we  still  complied  with  Level  2  .  Additional  internal  assessments  were 
performed  on  selected  Level  3  process  areas  to  determine  our  progress  to  Level  3  and  to  con¬ 
tinue  to  ensure  that  the  organization  was  ready  for  a  formal  external  assessment  for  Level  3  in 
1997.  We  received  our  Level  3  rating  from  an  external  assessment  in  September  1997. 
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We  have  found  that  internal  assessments  are  invaluable  at  our  organization.  In  1997,  it  was 
determined  after  our  external  assessment  that  we  had  a  4.1  to  1  return  on  investment  that 
year.  Internal  assessments  educate  new  project  personnel  about  the  CMM  and  the  assessment 
method.  They  motivate  change  and  verify  compliance  to  the  CMM  on  a  project-by-project 
basis.  We  also  found  internal  assessments  to  be  an  effective  tool  for  verifying  compliance 
after  reorganizations  and  readiness  for  ratings. 
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3  Question-and-Answer  Session 


The  audience  participated  in  a  question-and-answer  session  following  the  panelists’  presen¬ 
tations.  The  following  questions  were  asked: 

Why  conduct  a  CBA IPI  assessment  when  an  organization  is  already  aware  of  its  weak¬ 
nesses?  Why  pay  the  cost  of  doing  a  formal  CBA  IPI  assessment — estimated  to  cost 
$100,000? 

Berggrert:  Many  times  the  organization  does  not  have  a  good  enough  understanding  of  the 
CMM  to  be  aware  of  its  weaknesses.  The  assessment  process  applies  a  scientific  approach  to 
comparing  the  existing  practices  to  the  CMM  and  interpreting  the  CMM  in  the  environment 
being  assessed.  Most  of  our  assessments  run  less  than  $45,000  (including  travel). 

Des  Rochettes:  Members  of  the  internal  SEPG  cannot  be  fully  unbiased,  especially  if  there  is 
management  pressure.  We  have  some  examples  of  organizations  that  thought  they  were  Level 
3  and  were  assessed  at  Level  1.  Also  sometimes  the  SEPG  is  more  pessimistic  than  realistic. 

Iredale:  First  of  all,  we  did  not  find  our  internal  assessments  to  cost  $100,000.  We  have 
seven  SEI-authorized  lead  assessors,  and  they  are  paid  whether  they  are  performing  assess¬ 
ments  or  any  other  SEPG  function  within  our  organization.  An  assessment  is  a  logical  part  of 
the  IDEAL  improvement  cycle;  it  provides  the  ability  to  look  jointly  into  the  mirror  and 
openly  agree  on  the  problems.  It  also  helps  maintain  continuous  focus  on  SPI  even  in  places 
where  sponsorship  is  light. 

Lavi:  Being  aware  of  the  weaknesses  is  not  enough  to  start  correcting  them.  Organizations 
that  are  just  starting  to  improve  their  process  need  a  public  demonstration  to  start  the  im¬ 
provement  and  buy-in  efforts,  although  other  less  expensive  methods  (like  SPA)  may  serve 
the  same  purpose.  Other  organizations  need  the  CBA  IPI  assessment  because  it’s  the  only 
reliable  way  to  determine  an  organization’s  CMM  level,  when  achievement  of  certain  CMM 
levels  is  defined  as  a  program  milestone. 

Taylor:  First  of  all,  we  did  not  find  our  internal  assessments  to  cost  $100,000.  We  have  six 
authorized  lead  assessors,  and  they  are  paid  whether  they  are  performing  assessments  or  any 
other  SEPG  function  within  our  organization.  The  only  additional  cost  is  the  time  that  project 
personnel  are  required  to  be  available.  That  is  only  for  interviews  and  presentations,  which  is 
less  than  four  hours  per  individual.  Weaknesses  may  be  known  by  some,  but  they  are  never  as 
visible  as  when  they  turn  up  in  outbriefs. 
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How  often  do  you  do  full  CBA IPI  assessments? 


Berggren:  In  the  lower  maturity  levels,  every  12-18  months. 

Des  Rochettes :  Full  assessments  are  done  for  rating  a  maturity  level.  (Some  years  ago,  we 
did  dummy  assessments  for  the  full  process,  but  obviously  the  ROI  was  low  compared  with 
using  mini-assessments.)  At  Thomson-CSF,  it  takes  an  average  of  30  months  for  an  organiza¬ 
tion  to  get  to  Level  2.  After  the  strategic  plan  is  approved,  it  takes  an  average  of  22  months 
for  a  Level  2  organization  to  be  ready  for  a  Level  3  on-site  assessment. 

Iredale:  Usually  once  a  year. 

Lavi:  Usually  once  in  two  to  three  years. 

Taylor:  When  we  were  striving  to  establish  our  process  improvement  effort,  we  did  them 
more  often  than  after  we  were  Level  2  working  towards  Level  3.  We  never  did  one  less  than 
six  months  after  another.  Depending  on  the  size  of  the  organization,  I  recommend  an  assess¬ 
ment  at  least  every  two  years. 


On  how  small  of  an  organization  is  it  feasible  to  perform  a  CBA  IPI  assessment? 
Berggren:  The  smallest  group  I  have  assessed  is  65  people. 

Des  Rochettes :  The  smallest  organization  that  had  a  full  assessment  had  40  to  50  software 
practitioners. 

Iredale:  The  smallest  group  I  have  assessed  is  30  people. 

Lavi:  In  IAI  the  smallest  organization  in  which  we  perform  CBA  IPI  assessments  has  about 
35  software  engineers.  I  think  a  CBA  EPI  assessment  is  feasible  in  an  organization  that  has  at 
least  30  software  engineers. 

Taylor:  I  have  performed  a  CBA  IPI  assessment  on  an  organization  as  small  as  150  people. 
They  thought  that  it  was  effective  and  they  are  doing  quite  well  in  process  improvement  to 
date. 
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How  is  it  possible  to  maintain  consistency  among  assessments  being  done  internation¬ 
ally? 

Berggren:  The  assessment  method  and  the  model  provide  the  consistency.  For  me  it  was  a 
little  more  difficult  to  get  a  quick  understanding  of  the  development  environment  in  a  country 
where  I  was  not  fluent  in  the  language. 

Des  Rochettes:  In  our  company  there  is  an  ongoing  mix  of  interventions  (e.g.,  full  assess¬ 
ments,  mini-assessments,  consulting  days,  reviews)  of  our  U.S.  and  internal  lead  assessors,  as 
well  as  the  team  members.  All  of  our  internal  lead  assessors  were  previously  team  members 
for  SPAs,  then  for  CBA IPI  assessments  with  our  two  current  U;S.  lead  assessors.  They  were 
trained  and  coached  by  the  U.S.  lead  assessors.  This  means  that  there  is  a  cross-  and  auto- 
benchmarking  between  the  way  we  all  do  the  job.  To  date,  everything  has  gone  smoothly  and 
coherently,  which  means  that  we  strongly  think  that  we  are  standardized  with  our  U.S.  lead 
assessors,  who  are  some  of  the  most  active  lead  assessors.  But  above  all,  one  should  never 
forget  that  if  the  CMM  is  covered  and  the  CBA  IPI  method  is  followed,  then  the  repeatability 
of  the  results  is  ensured. 

Iredale:  We  try  to  use  a  core  team  of  seven  SEI-authorized  lead  assessors  who  have  all  been 
trained  at  Reuters  in  addition  to  their  SEI  training. 

Lavi:  Basically  a  CBA  IPI  assessment  is  an  internal  audit,  unlike  International  Organization 
for  Standardization  (ISO)  9000  audits  where  consistency  is  much  more  important.  Yet,  the 
CBA  IPI  method  is  very  well  defined,  and  this  contributes  to  the  consistency  and  reliability  of 
the  results.  In  addition  we  at  IAI  sometimes  call  on  external  lead  assessors  to  participate  in 
our  assessments,  although  we  have  internal  lead  assessors,  to  ensure  consistency  and  reliabil¬ 
ity  of  the  assessment  results. 


How  is  it  possible  to  reduce  the  impact,  stress,  and  disruption  on  an  organization  when 
an  assessment  is  conducted? 

Berggren:  The  key  is  planning  and  preparation.  The  sponsor  needs  to  communicate  to  the 
organization  that  an  assessment  is  going  to  be  done,  the  organization’s  expected  behavior,  the 
priority  of  the  assessment,  why  it  is  being  done,  and  what  will  be  done  with  the  results. 
Communicating  the  priority  of  the  assessment  to  the  organization  should  help  prevent  situa¬ 
tions  where  the  team  is  bumped  from  an  interview  room  or  is  interrupted  during  an  interview. 
A  good  site  coordinator  is  crucial  to  a  smooth-running  assessment.  He  or  she  must  communi¬ 
cate  the  schedule  and  logistics  to  the  participants  so  that  they  can  plan  accordingly.  The  site 
coordinator  needs  to  be  trained  in  the  method  as  soon  as  possible  to  begin  performing  this 
role.  Participants  need  to  adjust  commitments  so  that  they  can  participate  fully.  As  a  lead 
assessor,  I  plan  my  schedule  so  that  I  can  deliver  the  participants’  briefing  to  give  them  an 
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opportunity  to  meet  me,  ask  questions,  and  express  concerns  they  may  have  before  the  on-site 
period. 

Des  Rochettes:  First  of  all,  the  full  assessment  will  require  no  more  than  seven  hours  for 
most  people  (mainly  the  software  project  managers  and  the  SEPG)  and  an  average  of  less 
than  six  hours  for  the  majority  of  the  participants.  One  of  the  key  features  of  the  CBA IPI 
method  is  to  make  people  feel  free  and  comfortable,  and  our  experience  is  that  buy-in  is  usu¬ 
ally  obtained.  So  there  may  be  stress  before  the  assessment,  but  rarely  are  people  disap¬ 
pointed.  The  only  case  of  disappointment  is  when  management  doesn’t  follow  on  or  sponsor 
the  improvement  actions. 

Iredale:  The  key  is  planning  and  preparation.  We  also  use  mini-assessments  and  a  scoring 
system  to  track  SPI  progress  very  closely.  There  are  very  few  surprises  during  a  formal  CBA 
IPI  assessment. 

Taylor:  I  think  that  it  is  important  to  set  the  context  for  the  assessment  early.  Let  everyone 
know  why  the  assessment  is  being  done  and  especially  what  is  expected  of  them.  The  sooner 
everyone  understands  the  assessment  process,  the  better  they  can  prepare.  I  have  found  that 
once  they  know  what  is  going  to  happen  and  how  small  the  impact  on  them  is  expected  to  be, 
the  easier  it  is  to  get  buy-in  from  them.  Also  it  is  the  lead  assessor’s  responsibility  to  set  the 
tone  for  the  assessment.  It  should  be  non-confrontational,  and  it  should  be  seen  as  a  collabo¬ 
ration  between  the  assessment  team  and  the  organization  towards  a  common  goal.  It  is  espe¬ 
cially  important  that  the  participants  being  interviewed  feel  comfortable,  not  intimidated. 


If  there  is  one  thing  that  you  could  change  about  CBA  IPI  assessments,  what  would  it 
be? 

Berggren:  Late  hours — although  they  are  not  as  late  as  they  used  to  be. 

Des  Rochettes:  Perhaps  work  on  the  means  to  provide  the  best  recommendations;  for  exam¬ 
ple,  provide  an  additional  (but  optional)  framework  and  method  for  interviews  and  queries  on 
concerns  and  business  objectives,  and  a  tracing  of  findings  to  actual  problems  and/or  non¬ 
performance  of  processes. 

Iredale:  Streamline  the  KPA  wall-chart  process. 

Taylor:  No  long  nights!  I  thought  tagging  notes  and  wall  charts  were  things  that  could  be  im¬ 
proved.  We  developed  a  tool  that  helps  us  automatically  tag  notes  and  perform  consolidation 
with  an  overhead  projector.  That  eliminates  the  need  for  wall  charts  and  enables  the  team  to 
use  the  same  room  for  interviewing  and  team  work,  since  we  don’t  need  to  litter  the  walls. 
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How  are  conflict-of-interest  matters  handled  within  an  organization  during  an  assess¬ 
ment,  such  as  SEPG  members  serving  as  assessment  team  members  who  have  had  a  ma¬ 
jor  role  in  instituting  the  process  improvement  activities? 

Berggren:  The  lead  assessor  should  be  able  to  spot  hidden  agendas  on  the  team  regardless  of 
the  source,  and  make  sure  that  the  team  stays  focused  on  interpreting  the  CMM  in  the  context 
of  the  software  development  organization  being  examined.  If  the  conflict  cannot  be  handled 
through  facilitation,  the  lead  assessor  also  can  reassign  KPAs — splitting  up  SEPG  members, 
or  assigning  them  to  KPAs  not  critical  to  the  target  maturity  level.  Although  I  have  never  had 
to  do  it,  the  method  also  allows  for  assessment  team  members  to  be  removed  from  the  team. 
The  SEPG  members  on  the  team  have  to  abide  by  the  same  method  as  everyone  else.  The 
entire  team  (including  the  lead  assessor)  reviews  everyone’s  work  and  must  come  to  consen¬ 
sus  on  what  they  have  seen  and  heard.  The  consensus  process  usually  works  out  most  of  the 
issues  that  individual  team  members  may  be  pursuing  at  the  outset  of  the  assessment.  SEPG 
members  usually  come  away  from  the  assessment  talking  about  how  much  they  learned  and 
are  really  excited  about  working  on  the  action  plan. 

Des  Rochettes:  We  have  addressed  that  in  Section  2.2.3  (Key  Planning  Issues);  there  is  only 
one  representative  of  the  organization;  the  projects  are  chosen  in  a  kick-off  meeting,  with  the 
corporate  software  director,  the  lead  assessor,  and  the  sponsor  and  management  of  the  as¬ 
sessed  unit.  Now  when  we  do  Level  3  and  higher  assessments,  the  functional  area  represen¬ 
tatives  (FARs)  are  randomly  chosen  for  each  FAR  sessions. 

Iredale:  Through  consensus  of  the  team. 

Lavi:  In  IAI,  SEPG  leaders  are  not  allowed  to  participate  as  assessment  team  members  in 
their  own  division  (though  they  are  welcome  to  participate  as  team  members  in  other  divi¬ 
sions).  The  same  principle  holds  for  other  persons  who  may  have  direct  personal  interest  in 
proving  their  own  success  or  any  other  possible  personal  agenda.  However,  conflicts  always 
exist,  and  it  is  the  lead  assessor’s  responsibility  to  identify  and  resolve  such  conflicts.  Usually 
the  best  way  to  avoid  and  eliminate  such  conflicts  is  by  proper  selection  of  team  members, 
proper  assessment  training,  and  appropriate  leadership. 

Taylor:  I  think  the  lead  assessor  should  be  leery  of  using  SEPG  members  on  the  team,  espe¬ 
cially  if  they  had  a  major  role  in  the  process  improvement  effort.  They  do  tend  to  disrupt  con¬ 
sensus  since  they  have  so  much  at  stake.  I  have  allowed  them  on  some  teams  depending  on 
their  personality  and  willingness  to  get  accurate  results  versus  a  score. 
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